Rathbone dilahirkan di Singapura, anak kepada ibubapa Amerika Randee Lynn (née Brauner) dan Monroe Jackson Rathbone IV.[4][5] His great-grandfather, Monroe Jackson Rathbone II, adalah pengurus syarikat Standard Oil di New Jersey, yang kemudiannya menjadi Exxon;[5] Rathbone is also distantly related to Civil War general Stonewall Jackson dan pelakon British Basil Rathbone.[6] Because of his father's job at Mobil Oil, Rathbone telah tinggal di tempat dari Indonesia ke Midland, Texas.[7] He started out in local theatre in Midland with the youth actors program "The Pickwick Players", initially doing musical theatre.[8] For his junior dan senior years of high school, Rathbone attended the Interlochen Arts Academy, a private school for the arts in Michigan, where he majored in acting.[7][8] After graduation, he had planned to go to the Royal Scottish Academy of Music dan Drama, but went instead to Los Angeles to experiment with film acting.[7]